Optimize GLM-5.2-MXFP4 SGLang Agentic Performance on MI355X - #2900
Conversation
…ng-opt-0908 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # perf-changelog.yaml
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…lang-opt-0908 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # perf-changelog.yaml
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding on the perf-changelog.yaml trailing-line removal, I checked two other AGENTS.md conventions for this diff: the --use-chat-template requirement for *_mtp.sh scripts calling run_benchmark_serving (this script is an agentic-coding benchmark that doesn't call run_benchmark_serving directly, so it doesn't apply), and the model.container == image rule for image bumps (this is a single-node config with no container key, so that multi-node rule doesn't apply either).
Extended reasoning...
This diff is a small, mechanical-looking change (image version bump plus a one-line default-policy revert) but it carries a reported finding: the removal of the pre-existing trailing whitespace-only line at the end of perf-changelog.yaml, which conflicts with AGENTS.md's explicit "append-only, byte-sensitive" invariant for that file. Since a reported finding exists, I did not restate it but instead checked adjacent conventions that could plausibly also be violated by this change — the --use-chat-template flag requirement and the model.container/image consistency rule — and confirmed both are inapplicable to these particular files (no run_benchmark_serving call in this agentic script; no container key in this single-node config entry). Given the outstanding inline finding, a human should still review before merge.
| - "Bump the Qwen3.5 FP8 B200 SGLang image from lmsysorg/sglang:v0.5.14-cu130 to lmsysorg/sglang:v0.5.19-cu130." | ||
| - "Tune the 8k/1k serving recipe with --mamba-full-memory-ratio 0.37 and --linear-attn-prefill-backend flashinfer; raise max-prefill-tokens and chunked-prefill-size from 16384 to 32768 and mem-fraction-static from 0.8 to 0.86." | ||
| - "Expand the TP8 and TP4/EP1 sweep coverage through concurrency 640." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2866 |
There was a problem hiding this comment.
🟡 (optional) The diff deletes the file's pre-existing trailing whitespace-only line (old file ended "...pull/2866\n \n", new ends "...pull/2866\n"), violating AGENTS.md's rule that perf-changelog.yaml is append-only/byte-sensitive: preserve all existing bytes and separator whitespace, append only at the tail. Fix: re-add the removed trailing line so no existing bytes are altered; only append the new entry above it, never edit or truncate the tail.
Extended reasoning...
Base file (at 4caeb31) ends with the last pr-link line followed by a line containing two spaces and a newline (confirmed via od -c). HEAD's perf-changelog.yaml drops that trailing line entirely (git diff shows a bare deletion with no matching addition at file end). AGENTS.md line 31 states the file is append-only and byte-sensitive and existing bytes/separator whitespace must be preserved, appends only at the tail — this diff instead mutates the tail by removing a byte sequence, which any tooling relying on byte-stable diffs/checksums against this file would flag or choke on, and sets precedent that trailing separator content can be silently dropped in future PRs.
Verification: normal (violates a documented non-negotiable invariant this diff introduces). Base perf-changelog.yaml at 4caeb31 ends with ...pull/2866\n \n — od -c on the base tail shows 2866 \n \n (a trailing line of two spaces plus newline after the final pr-link). HEAD's file ends ...pull/2866\n — od -c on the current tail shows 2866 \n with the whitespace-only line gone. The diff's last…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34286209780 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
5 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
Cold mha_varlen compile after ready is ~40s; the default 20s generate health check marked the server UnHealthy and lm_eval hit a dead port. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed The eval-only GSM8K job on |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34312259596 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 81d4111. Configure here.
Co-authored-by: Cursor <cursoragent@cursor.com>
…lang-opt-0908 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # benchmarks/single_node/agentic/glm5.2_fp4_mi355x_sglang_mtp.sh
Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34572993517 |
sgl-project/sglang#30575 exposes the Fast Triton Sparse MLA path as an explicit backend; the 20260910 image already contains it. Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34573979027 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34582654120 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34582654120 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34582654120 |
|
/stage-results 34582654120 |
|
@chunfangamd staged run 34582654120: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-11~r34582654120 This run remains available across future |
|
/reuse-sweep-run 34582654120 |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34582654120
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34582654120
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
sgl-project/sglang#39106
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd — one blocker: the published SGLang cookbook recipe for GLM-5.2 MI355X MXFP4 still does not cover this PR's major parallelism args. The linked upstream PR (sgl-project/sglang#39106, merged 2026-09-11) only switches the DSA backend to triton; the MXFP4 cells remain TP4 with no ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
OK, Will update that part. |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34582654120
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34582654120
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
sgl-project/sglang#39230
Signed: @chunfangamd
✅✅✅ Verdict: PASS ✅✅✅Re-verification after the earlier REJECTED verdict on this same head: the Check 3 blocker is resolved by the newly merged cookbook update linked in @chunfangamd's latest sign-off (approved review, 2026-09-12 23:47 UTC). The conversation-comment id this run was triggered with (5649477596) returns 404, so the sign-off body was verified from that review. ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
| - config-keys: | ||
| - glm5.2-fp4-mi355x-sglang-agentic-mtp | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Update the SGLang ROCm image from lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728 to lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260910." | ||
| - "Pick up our recent SGLang main-branch optimizations for GLM-5.2-MXFP4 serving." | ||
| - "Lower the TP HiCache host-pool ratio from 1.5 to 1.0 (DP-attention arm stays at 0.5). Eval and sweep still share the same ratio-based sizing; the smaller pool cuts pinned host DRAM on the TP4 arm (~453 GB vs ~679 GB). Keep the write policy at write_through: evals-only run 34569310922 passed with ratio 1.0 and write_through, so write_through_selective was unrelated to the earlier failure." | ||
| - "Remove the SGLANG_HEALTH_CHECK_TIMEOUT=120 override and its stale first-request JIT explanation; successful evals-only run 34569310922 used the default health-check timeout." | ||
| - "Replace the removed decode-only alias --cuda-graph-max-bs with its canonical name --cuda-graph-max-bs-decode, retaining the same min(2*CONC, 64) cap. The 20260910 image includes sgl-project/sglang#38375, so argparse interprets the removed spelling as an ambiguous prefix of the decode and prefill options and exits before serving. Leave --cuda-graph-max-bs-prefill unset: it controls a separate prefill token-capture ceiling, not request concurrency." | ||
| - "Switch DSA prefill and decode from tilelang to triton (sgl-project/sglang#30575 Fast Triton Sparse MLA). The 20260910 image already contains that backend; kv-cache-dtype stays fp8_e4m3, which the Triton DSA path requires." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2900 |
There was a problem hiding this comment.
Could you move this PR’s changelog entry to the end during the final sync with main, as required by the append only policy?
https://github.com/SemiAnalysisAI/InferenceX/blob/main/CONTRIBUTING.md#L19
✅✅✅ Verdict: PASS ✅✅✅Verified at head ✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
5 participants
Description / 描述
English
This PR updates and tunes the GLM-5.2-MXFP4 SGLang AgentX configuration for MI355X:
lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728tolmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260910.tilelangto the upstream Fast Triton Sparse MLA backend.--cuda-graph-max-bsalias with the canonical--cuda-graph-max-bs-decodeoption while retaining themin(2 * CONC, 64)limit.1.5to1.0, lowering pinned host memory from approximately 679 GB to 453 GB for TP4. The DP-attention ratio remains0.5.write_throughas the default HiCache write policy.The PR does not patch or modify the inference engine inside the container; the pinned upstream image runs as shipped.
中文
本 PR 更新并优化 MI355X 上的 GLM-5.2-MXFP4 SGLang AgentX 配置：
lmsysorg/sglang-rocm:v0.5.16-rocm720-mi35x-20260728更新到lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260910。tilelang切换到上游 Fast Triton Sparse MLA 后端。--cuda-graph-max-bs-decode替换已移除的--cuda-graph-max-bs别名，同时保留min(2 * CONC, 64)的限制。1.5降低到1.0，使 TP4 的 pinned host memory 从约 679 GB 降至 453 GB；DP-attention ratio 保持为0.5。write_through为默认 HiCache 写入策略。perf-changelog.yaml中追加对应的性能变更记录。本 PR 不会修改或修补容器内的推理引擎；固定的上游镜像将按原样运行。
Related Issue / 相关事项
Supersedes #2887.
Type of Change / 变更类型
Validation / 验证
Validated configurations:
1, 2, 4, 8, 10, 12.1, 2, 4, 10.12.SGLANG_SIMULATE_ACC_LEN=3.61matches the committed thinking-enabled GLM-5.2 golden acceptance length for five-step MTP.Checklist / 检查清单
perf-changelog.yamlwithout modifying historical entries./reuse-sweep-run 34582654120after the final sweep and eval completed successfully.